15 research outputs found
Single-Microphone Speech Enhancement and Separation Using Deep Learning
The cocktail party problem comprises the challenging task of understanding a
speech signal in a complex acoustic environment, where multiple speakers and
background noise signals simultaneously interfere with the speech signal of
interest. A signal processing algorithm that can effectively increase the
speech intelligibility and quality of speech signals in such complicated
acoustic situations is highly desirable. Especially for applications involving
mobile communication devices and hearing assistive devices. Due to the
re-emergence of machine learning techniques, today, known as deep learning, the
challenges involved with such algorithms might be overcome. In this PhD thesis,
we study and develop deep learning-based techniques for two sub-disciplines of
the cocktail party problem: single-microphone speech enhancement and
single-microphone multi-talker speech separation. Specifically, we conduct
in-depth empirical analysis of the generalizability capability of modern deep
learning-based single-microphone speech enhancement algorithms. We show that
performance of such algorithms is closely linked to the training data, and good
generalizability can be achieved with carefully designed training data.
Furthermore, we propose uPIT, a deep learning-based algorithm for
single-microphone speech separation and we report state-of-the-art results on a
speaker-independent multi-talker speech separation task. Additionally, we show
that uPIT works well for joint speech separation and enhancement without
explicit prior knowledge about the noise type or number of speakers. Finally,
we show that deep learning-based speech enhancement algorithms designed to
minimize the classical short-time spectral amplitude mean squared error leads
to enhanced speech signals which are essentially optimal in terms of STOI, a
state-of-the-art speech intelligibility estimator.Comment: PhD Thesis. 233 page
Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks
In this paper we propose the utterance-level Permutation Invariant Training
(uPIT) technique. uPIT is a practically applicable, end-to-end, deep learning
based solution for speaker independent multi-talker speech separation.
Specifically, uPIT extends the recently proposed Permutation Invariant Training
(PIT) technique with an utterance-level cost function, hence eliminating the
need for solving an additional permutation problem during inference, which is
otherwise required by frame-level PIT. We achieve this using Recurrent Neural
Networks (RNNs) that, during training, minimize the utterance-level separation
error, hence forcing separated frames belonging to the same speaker to be
aligned to the same output stream. In practice, this allows RNNs, trained with
uPIT, to separate multi-talker mixed speech without any prior knowledge of
signal duration, number of speakers, speaker identity or gender. We evaluated
uPIT on the WSJ0 and Danish two- and three-talker mixed-speech separation tasks
and found that uPIT outperforms techniques based on Non-negative Matrix
Factorization (NMF) and Computational Auditory Scene Analysis (CASA), and
compares favorably with Deep Clustering (DPCL) and the Deep Attractor Network
(DANet). Furthermore, we found that models trained with uPIT generalize well to
unseen speakers and languages. Finally, we found that a single model, trained
with uPIT, can handle both two-speaker, and three-speaker speech mixtures
Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation
We propose a novel deep learning model, which supports permutation invariant
training (PIT), for speaker independent multi-talker speech separation,
commonly known as the cocktail-party problem. Different from most of the prior
arts that treat speech separation as a multi-class regression problem and the
deep clustering technique that considers it a segmentation (or clustering)
problem, our model optimizes for the separation regression error, ignoring the
order of mixing sources. This strategy cleverly solves the long-lasting label
permutation problem that has prevented progress on deep learning based
techniques for speech separation. Experiments on the equal-energy mixing setup
of a Danish corpus confirms the effectiveness of PIT. We believe improvements
built upon PIT can eventually solve the cocktail-party problem and enable
real-world adoption of, e.g., automatic meeting transcription and multi-party
human-computer interaction, where overlapping speech is common.Comment: 5 page
On the Relationship Between Short-Time Objective Intelligibility and Short-Time Spectral-Amplitude Mean-Square Error for Speech Enhancement
The majority of deep neural network (DNN) based speech enhancement algorithms
rely on the mean-square error (MSE) criterion of short-time spectral amplitudes
(STSA), which has no apparent link to human perception, e.g. speech
intelligibility. Short-Time Objective Intelligibility (STOI), a popular
state-of-the-art speech intelligibility estimator, on the other hand, relies on
linear correlation of speech temporal envelopes. This raises the question if a
DNN training criterion based on envelope linear correlation (ELC) can lead to
improved speech intelligibility performance of DNN based speech enhancement
algorithms compared to algorithms based on the STSA-MSE criterion. In this
paper we derive that, under certain general conditions, the STSA-MSE and ELC
criteria are practically equivalent, and we provide empirical data to support
our theoretical results. Furthermore, our experimental findings suggest that
the standard STSA minimum-MSE estimator is near optimal, if the objective is to
enhance noisy speech in a manner which is optimal with respect to the STOI
speech intelligibility estimator
On Loss Functions for Supervised Monaural Time-Domain Speech Enhancement
Many deep learning-based speech enhancement algorithms are designed to
minimize the mean-square error (MSE) in some transform domain between a
predicted and a target speech signal. However, optimizing for MSE does not
necessarily guarantee high speech quality or intelligibility, which is the
ultimate goal of many speech enhancement algorithms. Additionally, only little
is known about the impact of the loss function on the emerging class of
time-domain deep learning-based speech enhancement systems. We study how
popular loss functions influence the performance of deep learning-based speech
enhancement systems. First, we demonstrate that perceptually inspired loss
functions might be advantageous if the receiver is the human auditory system.
Furthermore, we show that the learning rate is a crucial design parameter even
for adaptive gradient-based optimizers, which has been generally overlooked in
the literature. Also, we found that waveform matching performance metrics must
be used with caution as they in certain situations can fail completely.
Finally, we show that a loss function based on scale-invariant
signal-to-distortion ratio (SI-SDR) achieves good general performance across a
range of popular speech enhancement evaluation metrics, which suggests that
SI-SDR is a good candidate as a general-purpose loss function for speech
enhancement systems.Comment: Published in the IEEE Transactions on Audio, Speech and Language
Processin